Solution and Forecast Horizons for Infinite-Horizon Nonhomogeneous Markov Decision Processes

We address in this paper the challenge of solving a nonhomogeneous infinite horizon Markov Decision Process (MDP) problem. More precisely, we seek an algorithm that, when given a finite subset of the problem’s potentially infinite data set, delivers an optimal first period policy. Such an algorithm can thus recursively generate within a rolling horizon procedure an infinite horizon optimal solution to the original infinite horizon problem. However it can happen that for a given problem no such algorithm exists. In this case, it is impossible to solve the problem with a finite state machine. We say such problems fail to be well-posed. Under the assumption of increasing marginal returns in actions with respect to states and stochastically increasing states transitioned into with respect to actions, we provide an algorithm that is guaranteed to solve the corresponding nonhomogeneous MDP whenever that problem is well-posed. The algorithm proceeds by discovering in finite time a forecast horizon for which a optimal solution delivers an optimal first period policy to the infinite horizon problem. In particular, we show by construction the existence of a forecast horizon (and hence a solution horizon) for all such well-posed problems. We illustrate the theory and algorithms developed by solving the time-varying version of the classic asset selling problem. 1991 Mathematics Subject Classification. Primary 90C40 Secondary 90B15, 90C39.